Deterministic convergence of conjugate gradient method for feedforward neural networks

Conjugate gradient methods have many advantages in real numerical experiments, such as fast convergence and low memory requirements. This paper considers a class of conjugate gradient learning methods for backpropagation (BP) neural networks with three layers. We propose a new learning algorithm for almost cyclic BP neural networks based on PRP conjugate gradient method. We then establish the deterministic convergence properties for three different learning fashions, i.e., batch mode, cyclic and almost cyclic learning. There are mainly two deterministic convergence properties including weak and strong convergence which indicate the gradient of the error function goes to zero and the weight sequence goes to a fixed point, respectively. Learning rate plays an important role in the training process of BP neural networks. The deterministic convergence results based on different learning fashions are dependent on different selection strategies of learning rate.